1. Field of the Invention
The present invention relates generally to a 2D/3D image conversion system for converting two-dimensional images into three-dimensional images.
2. Related Art
There has been known a method of converting two-dimensional (2D) images into three-dimensional (3D) images including the steps of: generating, from a 2D image signal, a main image signal and a sub-image signal time-delayed relative to the main image signal by utilizing a plurality of field memories; and outputting either one of the main and sub-image signals as a left-eye image signal and the other as a right-eye image signal.
A time delay of the sub-image signal relative to the main image signal (hereinafter referred to as xe2x80x9cdelay valuexe2x80x9d) is determined based on a velocity of horizontal motion of the main image. That is, the greater the velocity of horizontal motion of the main image, the smaller the delay value. Further, which of the main image and the sub-image is represented by the left-eye image signal with the other represented by the right-eye image signal is determined based on a direction (leftward or rightward) of the horizontal motion of the main image.
Since such a method is adapted to convert only 2D moving images into 3D images, this method is referred to as xe2x80x9c2D/3D moving image conversion methodxe2x80x9d.
The present applicants have already developed a method of converting 2D still images into 3D images (hereinafter referred to as xe2x80x9c2D/3D still image conversion methodxe2x80x9d), having filed an application for patent (Japanese Patent Application No.9(1997)-159949). However, having not yet been laid open, this application does not constitute the prior art.
This 2D/3D still image conversion method includes the steps of: extracting from a 2D input image signal a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, the extraction performed on a field-by-field basis; generating parallax information per given unit area in the one-field screen based on the image characteristic value per parallax calculation region thus extracted; and generating a first image signal and a second image signal from a signal residing in each of the given unit areas of the 2D input image, the first and second image signals having a horizontal phase difference therebetween based on the parallax information corresponding to the given unit area.
Now referring to FIGS. 1 to 3, description will be made on the 2D/3D still image conversion method (Japanese Patent Application No.9(1997)-159949) developed by the applicants. It is to be noted that this 2D/3D still image conversion method has not yet been known to the art and hence, does not constitute the prior art.
FIG. 1 is a block diagram showing a whole construction of a 2D/3D image conversion system for converting 2D images into 3D images.
A luminance signal Y and color difference signals R-Y and B-Y constituting a 2D image signal are respectively converted into digital signals Y, R-Y and B-Y by means of an AD conversion circuit 1 (ADC).
The Y signal is supplied to a high-frequency component integrating circuit 8 and a luminance contrast calculating circuit 9 as well as to a first left-eye-image optional pixel delay FIFO 11 and a first right-eye-image optional pixel delay 21. The R-Y signal is supplied to an R-Y component integrating circuit 31 as well as to a second left-eye-image optional pixel delay FIFO 12 and a second right-eye-image optional pixel delay FIFO 22. The B-Y signal is supplied to a B-Y component integrating circuit 32 as well as to a third left-eye-image optional pixel delay FIFO 13 and a third right-eye-image optional pixel delay FIFO 23.
As shown in FIG. 2, the high-frequency component integrating circuit 8 performs calculation on a field-by-field basis for giving an integrated value of high-frequency component of each of the plural parallax calculation regions E1 to E12 previously defined in the one-field screen. The luminance contrast calculating circuit 9 calculates a luminance contrast of each of the parallax calculation regions E1 to E12 on a field-by-field basis. The R-Y component integrating circuit 31 calculates an integrated value of R-Y component of each of the parallax calculation regions E1 to E12 on a field-by-field basis. The B-Y component integrating circuit 32 calculates an integrated value of B-Y component of each of the parallax calculation regions E1 to E12 on a field-by-field basis.
The high-frequency component integration values, luminance contrasts, R-Y component integrated values and B-Y component integrated values of the respective parallax calculation regions E1 to E12 are used as perspective image characteristic values of the respective parallax calculation regions E1 to E12.
In an example shown in FIG. 13, the one-field screen includes a total of 60 parallax calculation regions in 6 rows and 10 columns. However, it is assumed for convenience in explanation that a total of 12 parallax calculation regions E1 to E12 in 3 rows and 4 columns are defined in a one-field screen, as shown in FIG. 2.
A CPU 3 generates parallax information on the respective parallax calculation regions E1 to E12 based on information supplied from the high-frequency component integrating circuit 8, luminance contrast calculating circuit 9, R-Y component integrating circuit 31 and B-Y component integrating circuit 32. In this example, the parallax information is generated such that the more to the front is located an object like a subject in a scene of the example, the smaller is the parallax value while the more to the back is located an object like a background of the scene of the example, the greater is the parallax value. A method of generating the parallax information will hereinafter be described in detail.
The parallax information per parallax calculation region E1 to E12 thus given by the CPU 3 is sent to a parallax control circuit 4. The parallax control circuit 4, in turn, generates parallax information per pixel position in each field based on the parallax information on the respective parallax calculation regions E1 to E12. Based on the parallax information per pixel position, the parallax control circuit 4 controls readout addresses of FIFOs 11 to 13 and 21 to 23 so as to read out an image signal(Y, R-Y and B-Y signals) from the left-eye-image optional pixel delays FIFOs 11 to 13 and from the right-eye-image optional pixel delays FIFOs 21 to 23, respectively, with the addresses of FIFOs 11 to 13 and of FIFOs 21 to 23 shifted from each other. Hence, left-eye image signals read out from the left-eye-image optional pixel delays FIFOs 11 to 13 have different horizontal phases from right-eye image signals read out from the right-eye-image optional pixel delays FIFOs 21 to 23.
The left-eye image signals (YL signal, (R-Y)L signal and (B-Y)L signal) read out from the left-eye-image optional pixel delays FIFOs 11 to 13 are converted into analog signals through a DA conversion circuit (DAC) 5 before supplied to an unillustrated three-dimensional display unit. The right-eye image signals (YR signal, (R-Y)R signal and (B-Y)R signal) read out from the right-eye-image optional pixel delays FIFOs 21 to 23 are converted into analog signals through a DA conversion circuit (DAC) 6 before supplied to the unillustrated three-dimensional display unit.
Since the left-eye image signal has a different horizontal phase from that of the right-eye image signal, a parallax is produced between the left eye image and the right eye image. Hence, by viewing the left eye image with the left eye alone and the right eye image with the right eye alone, a three-dimensional image is established wherein the subject is located to the front against the background.
FIG. 3 diagrammatically illustrates a configuration of the R-Y component integrating circuit 31.
FIG. 2 shows horizontal positions (HAD) and vertical positions (VAD) wherein a number of horizontal pixels of each parallax calculation region E1 to E12 is represented by xe2x80x9cmxe2x80x9d, that of vertical pixels of each parallax calculation regions is represented by xe2x80x9cnxe2x80x9d and coordinates of an upper left vertex of the first parallax calculation region E1 are represented by (a, b).
The R-Y component integrating circuit 31 includes a timing signal generating circuit 201, an adder circuit 202, an R-Y component integration register group 203 and the selection circuit (SEL) 204. The R-Y component integration register group 203 includes first to twelfth R-Y component integration registers 211 to 222 corresponding to the respective parallax calculation regions E1 to E12.
The timing signal generating circuit 201 is supplied with a horizontal synchronizing signal Hsync and a vertical synchronizing signal Vsync of an input image signal, and a clock signal CLK for detection of a horizontal address of each horizontal period.
The timing signal generating circuit 201 serves to output first to twelfth enable signals EN1 to EN12, a reset signal RST and an output timing signal DOUT based on the horizontal synchronizing signal Hsync, vertical synchronizing signal Vsync and clock signal CLK.
The enable signals EN1 to EN12 correspond to the parallax calculation regions E1 to E12, respectively, and are normally at L level, rising to H level when the horizontal/vertical position of the input image signal is in a region corresponding thereto. The first to twelfth enable signals EN1 to En12 are inputted in first to twelfth luminance integration registers 211 to 222 as a write signal, respectively. The first to twelfth enable signals EN1 to EN12 are supplied to a selection circuit 204, as well. The selection circuit 204 selectively outputs input data in correspondence to an enable signal at H level.
The reset signal RST is outputted as timed to a start of a valid image per field of the input image signal so as to be supplied to the respective R-Y component integration registers 211 to 222. With input of the reset signal RST, a content of the respective R-Y component integration registers 211 to 222 is reset to 0.
As shown in FIG. 2, the timing signal DOUT is at H level during a predetermined period of time starting from when the vertical position of the input image signal passes a vertically lowermost position of the parallax calculation region E12 in the lowest row. The output timing signal DOUT is supplied to the CPU 3.
The reset signal is outputted as timed to the start of the valid image in the input image signal for resetting the contents of the respective R-Y component integration registers 211 to 222 to 0. Where the horizontal/vertical position of the input image signal is in the first parallax calculation region E1, the first enable signal EN1 rises to H level, so that an R-Y value retained by the first R-Y component integration register 211 is supplied to the adder circuit 202 via the selection circuit 204 while the R-Y signal of the input image signal is applied to the adder circuit 202.
Accordingly, the R-Y value retained by the first luminance integration register 211 and the R-Y signal of the input image signal are added together by the adder circuit 202. A resultant sum is stored in the first R-Y component integration register 211. That is, when the horizontal/vertical position of the input image signal is in the first parallax calculation region E1, R-Y values of the pixels in the first parallax calculation region E1 are sequentially integrated while the integration results are accordingly stored in the first R-Y component integration register 211.
In this manner, the R-Y component integration values of the respective parallax calculation regions E1 to E12 are stored in the corresponding R-Y component integration registers 211 to 222. When the output timing signal DOUT rises to H level, the R-Y component integration values stored in the respective R-Y component integration registers 211 to 222 in correspondence to the parallax calculation regions E1 to E12 are supplied to the CPU 3 via a data bus (DATA-BUS).
The B-Y component integrating circuit 32 has the same configuration with the R-Y component integrating circuit 31 of FIG. 3 and therefore, a description thereof is omitted.
FIG. 4 diagrammatically illustrates a configuration of the high-frequency component integrating circuit 8. The high-frequency component integrating circuit 8 includes a timing signal generating circuit 231, a high-pass filter (HPF) 232, an absolutizing circuit 233, a slicing circuit 234, an adder circuit 235, a high-frequency component integration register group 236 and a selection circuit 237. The high-frequency component integration register group 236 includes first to twelfth high-frequency component integration registers 241 to 252 in correspondence to the parallax calculation regions E1 to E12.
Input and output signals of the timing signal generating circuit 231 are the same with those of the timing signal generating circuit 201 shown in FIG. 3.
As shown in FIG. 5, for example, the high-pass filter 232 may include five D flip-flops 261 to 165, a bit shift circuit 266 for providing an output twice the value of an input value, an adder 267 and a subtractor 268, having tap factors of xe2x88x921, 0, 2, 0 and xe2x88x921.
The slicing circuit 234 may have input/output characteristics shown in FIG. 6. The circuit is adapted to provide an output of 0 with respect to an input in a range of between 0 and Ia in order to prevent noises from being extracted as a high-frequency component.
Thus, the high-pass filter 232 extracts a high-frequency component of Y signal of the input image signal, an absolute value of which component is obtained by the absolutizing circuit 233 and is removed of noises by the slicing circuit 234.
The reset signal is outputted as timed to the start of a valid image of the input image signal so as to reset contents of the high-frequency component integration registers 241 to 252 to 0. Where a horizontal/vertical position of the input image signal is in the first parallax calculation region E1, the first enable signal EN1 rises to H level so that the high-frequency component retained in the first high-frequency component integration register 241 is supplied to the adder circuit 235 via the selection circuit 237 while the high-frequency component of Y signal (or the output from the slicing circuit 234) of the input image signal is supplied to the adder circuit 235.
Thus, the high-frequency component retained by the first high-frequency component integration register 241 and that of Y signal of the input image signal are added together by the adder circuit 235. A resultant sum is stored in the first high-frequency component integration register 241. That is, when the horizontal/vertical position of the input image signal is in the first parallax calculation region E1, high-frequency components of the pixels in the first parallax calculation region E1 are sequentially integrated while integration results are accordingly stored in the first high-frequency component integration register 241.
In this manner, high-frequency component integration values of the respective parallax calculation regions E1 to E12 are stored in the corresponding high-frequency component integration registers 241 to 252. When the output timing signal DOUT rises to H level, the high-frequency component integration values stored in the respective high-frequency component integration registers 241 to 252 in correspondence to the parallax calculation regions E1 to E12 are supplied to the CPU 3 via a data bus.
FIG. 7 diagrammatically illustrates another exemplary configuration of the high-frequency component integration circuit 8.
This high-frequency component integration circuit 8 includes a timing signal generating circuit 238, the high-pass filter 232, a peak detecting circuit 239, the adder circuit 235, the high-frequency component integration register group 236 and the selection circuit 237.
The timing signal generating circuit 238 operates substantially the same way with the timing signal generating circuit 201 of FIG. 3 except for that a trigger pulse (region boundary signal RST1) is outputted when the input image signal reaches each horizontal position preceding the parallax calculation region E1, E5 or D9 and the last horizontal position of each parallax calculation region E1 to E12, as shown in FIG. 2. The region boundary signal RST1 is supplied to the peak detecting circuit 239.
The high-frequency component of Y signal extracted by the high-pass filter 232 is supplied to the peak detecting circuit 239. The peak detecting circuit 239 detects a maximum value of the high-frequency components of each horizontal line of each parallax calculation region E1 to E12. The peak detecting circuit 239 may include a comparison circuit 271, a maximum value register 272 and a gate 273, as shown in FIG. 8. FIG. 9 is a timing chart showing the horizontal synchronizing signal Hsync, the region boundary signal RST1 and an output from the gate 273.
The maximum value register 272 is supplied with the high-frequency component of Y signal extracted by the high-pass filter 232, the region boundary signal RST1, a judgment signal La from the comparison circuit 271 and the clock signal CLK. The comparison circuit 271 compares the output from the maximum value register 272 and the high-frequency component of Y signal of the input image signal so as to raise the judgment signal La to H level when determining the high-frequency component of Y signal to be greater than the output from the maximum value register 272.
Rising to H level, the region boundary signal RST1 resets a content of the maximum value register 272 to 0. If the judgment signal La from the comparison circuit 271 is at H level while the region boundary signal RST1 is at L level, the high-frequency component of Y signal is stored in the maximum value register 272. That is, the content of the maximum value register 272 is updated. Thus, in each period during which the region boundary signal RST1 is at L level, the maximum value register 272 stores the maximum value of high-frequency components of Y signal with respect to the respective pixels in one horizontal line of one of the parallax calculation regions E1 to E12 that corresponds to the horizontal/vertical position of the input image signal.
The gate 273 provides an output value of the maximum value register 272 in response to the region boundary signal RST1 rising to H level while providing an output of 0 when the region boundary signal RST1 is at L level. More specifically, each time the region boundary signal RST1 rises to H level, the gate circuit 273 outputs the maximum value of the high-frequency components of Y signal with respect to one horizontal line of a given parallax calculation region E1 to E12, the maximum value stored in the maximum value register 272. Hence, integration values of maximum values of high-frequency components of Y signal with respect to the respective horizontal lines of the respective parallax calculation regions are stored in the corresponding high-frequency integration registers 241 to 252 (see FIG. 7).
FIG. 10 diagrammatically illustrates a configuration of the luminance contrast calculating circuit 9.
The luminance contrast calculating circuit 9 includes a timing signal generating circuit 301 and a luminance contrast sensing circuit group 302. The luminance contrast sensing circuit group 302 includes first to twelfth luminance contrast sensing circuits 311 to 322 corresponding to the parallax calculation regions E1 to E12, respectively.
The timing signal generating circuit 301 provides the same input and output signals as those of the timing signal generating circuit 201 shown in FIG. 3.
As shown in FIG. 11, the luminance contrast sensing circuits 311 to 322 each include a first comparison circuit 331, a maximum value register 332, a second comparison circuit 333, a minimum value register 334 and a subtractor 335.
The maximum value register 332 is supplied with Y signal of the input image signal, one of the enable signals EN (N=1,2 . . . 12) associated with one of the regions E1 to E12 corresponding to one of the luminance contrast sensing circuits, a reset signal RST, a judgment signal Lb outputted from the first comparison circuit 331 and a clock signal CLK. The first comparison circuit 331 compares an output value of the maximum value register 332 with Y signal of the input image signal so as to raise the judgment signal Lb to H level when determining the Y signal of the input image signal to be greater than the output value from the maximum value register 332.
Raised to H level, the reset signal RST resets a content of the maximum value register 332 to 0. When the enable signal EN associated with that of the regions E1 to E12 corresponding to the luminance contrast sensing circuit and the judgment signal Lb are both at H level, the Y signal is stored in the maximum value register 332. That is, the content of the maximum value register 332 is updated. Accordingly, immediately before an output timing signal DOUT is outputted, the maximum value of luminance values of the respective pixels in that of the parallax calculation regions E1 to E12 corresponding to the luminance contrast sensing circuit is stored in the maximum value register 332.
The minimum value register 334 is supplied with Y signal of the input image signal, an enable signal EN (N=1,2 . . . 12) associated with one of the regions E1 to E12 corresponding to one of luminance contrast sensing circuits, the reset signal RST, a judgment signal Lc outputted from the second comparison circuit 333 and the clock signal CLK. The second comparison circuit 333 compares an output value from the minimum value register 334 with the Y signal of the input image signal so as to raise the judgment signal Lc to H level when determining the Y signal of the input image signal to be smaller than the output value from the minimum value register 334.
When the reset signal RST rises to H level, a predetermined maximum value is set in the minimum value register 334. When the enable signal EN associated with that of the regions E1 to E12 corresponding to the luminance contrast sensing circuit and the judgment signal Lc are both at H level, the Y signal is stored in the minimum value register 334. That is, the content of the minimum value register 334 is updated. Accordingly, immediately before the output timing signal DOUT is outputted, the minimum value of luminance values of the pixels in that of the parallax calculation regions E1 to E12 corresponding to the luminance contrast sensing circuit is stored in the minimum value register 334.
As a result, at the time of output of the output timing signal DOUT, an output from the subtractor 335 has a value equal to a difference (luminance contrast) between the maximum value and the minimum value of the luminance values of the pixels in the corresponding one of the parallax calculation regions E1 to E12. Subsequently, when the output timing signal DOUT is outputted, the subtractor 335 applies the output (luminance contrast) to the CPU 3.
FIG. 12 is a flow chart representing steps in a parallax information generating procedure taken by the CPU 3 on a per-parallax-calculation-region basis.
The parallax information generating procedure performed on each segment region includes: a grouping processing (Step 1); a spatial separation processing (Step 2); a singular point processing (Step 3); an intergroup coupling processing (Step 4); a per-group depth information generation processing (Step 5); an all-region depth information correction processing (Step 6); a group-boundary depth information correction processing (Step 7); an in-group depth information correction processing (Step 8); and a parallax information calculation processing (Step 9).
Now the parallax information generating procedure will hereinbelow be described by way of example of 60 parallax calculation regions actually defined in one field. FIG. 13 shows 60 parallax calculation regions F1 to F60 defined in one field.
(1) Grouping Processing
The grouping processing of Step 1 is an initial processing of the procedure which is intended to divide all the regions constituting one image frame into groups associated with respective objects included in the image frame. There are two grouping methods which will be described as below.
(1-1) First Method
First, high-frequency component integration values of the respective parallax calculation regions F1 to F60 are normalized to values in a predetermined range (e.g., 0 to 20). This is followed by producing a distribution (histogram) of parallax calculation regions (by number) belonging to the respective values normalized from the high-frequency component integration values. FIG. 14 shows an example of the histogram thus produced. Parallax calculation regions included in bars between valleys in the histogram are combined into one group. The histogram may be based on luminance contrasts instead of high-frequency component integration values. FIG. 15 shows the result of dividing the parallax calculation regions F1 to F60 into groups with symbols G1 to G4 each indicating a group number.
(1-2) Second Method
First, R-Y component integration values of the respective parallax calculation regions F1 to F60 are normalized to values in a range of between 0 and 20. This is followed by the production of distribution (histogram) of parallax calculation regions (by number) belonging to the respective normalized values of the R-Y component integration values. Based on the resultant histogram, intergroup boundary values are found from the normalized values of the R-Y component integration values.
On the other hand, the B-Y component integration values of the respective parallax calculation regions F1 to F60 are normalized to values in a range of between 0 and 10. This is followed by the production of distribution (histogram) of parallax calculation regions (by number) belonging to the respective normalized values of the B-Y component integration values. Based on the resultant histogram, intergroup boundary values are found from the normalized values of the B-Y component integration values.
By using two types of boundary values thus found, all the parallax calculation regions F1 to F60 are divided into groups, as shown in FIG. 16. FIGS. 17 and 18 show the result of dividing the parallax calculation regions F1 to F60 into groups with symbols G1 to G5 each indicating a group number.
In this example, the grouping processing is performed according to the second method.
(2) Spatial Separation Processing In the spatial separation processing of Step 2, out of the parallax calculation regions combined into the same group at Step 1, parallax calculation regions spatially adjoining each other are considered to belong to the same group. In short, although combined into the same group by the processing of Step 1, parallax calculation regions spatially separated by another group are considered to belong to different groups.
More specifically, the parallax calculation regions determined to be included in the group 3 (G3) are divided into three groups 31 (G31), 32 (G32) and 33 (G33).
(3) Singular Point Processing
In a case where a group consisting of a single parallax calculation region exists, the singular point processing determines whether the single parallax calculation region corresponds to an object different from object(s) corresponded by other adjoining groups or to the same object with that corresponded by other adjoining group.
It is assumed, for example, that a group consists of a parallax calculation region A alone, as shown in FIG. 20. In the figure, of the two parallax calculation regions located upwardly of the parallax calculation region A, the closest one to the region A is denoted by U1 and the other by U2. Of the two parallax calculation regions located downwardly of the region A, the closest one to the region A is denoted by D1 and the other by D2. Of the two parallax calculation regions located leftwardly of the region A, the closest one to the region A is denoted by L1 and the other by D2. Of the two parallax calculation regions located rightwardly of the region A, the closest one to the region A is denoted by R1 and the other by R2.
In this case, when color distances between the region A and the adjoining regions U1, D1, L1 and R1 are greater than those between the regions U1, D1, L1 and R1 and their adjoining regions U2, D2, L2 and R2, respectively, the group consisting of the region A alone is determined to constitute one independent group. Otherwise, the region A is determined to belong to any one of the neighboring groups. That is, the grouping of the regions is corrected.
Now, the definition of the color distance will be described. It is herein assumed that a B-Y component integration value and an R-Y component integration value with respect to a certain parallax calculation region Fa are denoted by Fa(B-Y) and Fa(R-Y) while a B-Y component integration value and an R-Y component integration value with respect to another parallax calculation region Fb are denoted by Fb(B-Y) and Fb(R-Y). Then, a color distance xe2x80x9cdistxe2x80x9d between the region Fa and the region Fb is defined by the following equation (1):
dist=|Fa(B-Y)xe2x88x92Fb(B-Y)|+|Fa(R-Y)xe2x88x92Fb(R-Y)|xe2x80x83xe2x80x83(1)
For example, if the region A has (B-Y component integration value, R-Y component integration value) of (xe2x88x924, 5), the region U2 has those of (xe2x88x925, 4) and the region U2 has those of (xe2x88x927, 2), in FIG. 20. Then, a color distance xe2x80x9cdistxe2x80x9d between the regions A and U1 is xe2x80x9c2xe2x80x9d whereas a color distance xe2x80x9cdistxe2x80x9d between the regions U1 and U2 is xe2x80x9c4xe2x80x9d.
If the group 31 (G31) in FIG. 19 consists of a single parallax calculation region and is determined to belong to the group 1 (G1) by the above singular point processing, the grouping of the regions is corrected as shown in FIG. 21.
(4) Intergroup Coupling Processing
In the intergroup coupling processing of Step 4, firstly calculated are mean values of the R-Y component integration values and of the B-Y component integration values of parallax calculation regions constituting each group.
Subsequently, a color distance between adjacent two groups is calculated. Provided that the two adjoining groups are denoted by Ga and Gb. If the group Ga consists of n parallax calculation regions a1, a2, . . . an, a mean value *Ga(B-Y) of the B-Y component integration values and a mean value *Ga(R-Y) of the R-Y component integration values of the group Ga are given by the following equations (2):
*Ga(B-Y)={a1(B-Y)+a2(B-Y)+ . . . +an(B-Y)}÷n *Ga(R-Y)={a1(R-Y)+a2(R-Y)+ . . . +an(R-Y)}÷nxe2x80x83xe2x80x83(2)
Provided that the group Gb consists of m parallax calculation regions b1, b2, . . . bm, a mean value *Gb(B-Y) of the B-Y component integration values and a mean value *Gb(R-Y) of the R-Y component integration values of the group Gb are given by the following equations (3):
*Gb(B-Y)={b1(B-Y)+b2(B-Y)+ . . . +bm(B-Y)}÷m *Gb(R-Y)={b1(R-Y)+b2(R-Y)+ . . . +bm(R-Y)}xe2x80x83xe2x80x83(3)
A color distance xe2x80x9cdistxe2x80x9d between the groups Ga and Gb is defined by the following equation (4):
dist=|*Ga(B-Y)xe2x88x92*Gb(B-Y)|+|*Ga(R-Y)xe2x88x92*Gb(R-Y)|xe2x80x83xe2x80x83(4)
Then, whether the color distance between the adjacent groups is smaller than a threshold value or not is determined. If the color distance is smaller than the threshold value, these two groups are coupled together or combined into one group.
(5) Per-Group Depth Information Generation Processing
In the per-group depth information generation processing of Step 5, high-frequency component integration values of the respective parallax calculation regions F1 to F60 are first normalized to values in a range of between 0 and 10. Further, luminance contrasts of the respective parallax calculation regions F1 to F60 are also normalized to values in the range of between 0 and 10.
The resultant normalized values of the high-frequency component integration values and of the luminance contrasts as well as background weight components previously given to the respective parallax calculation regions F1 to F60 are used to generate depth information on a group-by-group basis.
Now, description will be made on a process of generating depth information on optional one group. First, the number n of parallax calculation regions included in this group is found. On the other hand, a calculation is performed to find a sum total xcexa3a of normalized values xe2x80x9caxe2x80x9d of high-frequency component integration values of the parallax calculation regions included in the group. A calculation is also performed to find a sum total xcexa3b of normalized values b of luminance contrasts of the parallax regions of the group. Further, a calculation is performed to find a sum total xcexa3c of weight components c with respect to the parallax calculation regions of the group.
Then, depth information H on the group is generated based on the following equation (5):
H=(K1xc2x7xcexa3a+K2xc2x7xcexa3b+K3xc2x7xcexa3c)÷nxe2x80x83xe2x80x83(5)
wherein K1, K2 and K3 denote factors defined as, for example, K1=3/8, K2=1/8, K3=4/8, respectively.
(6) All-Region Depth Information Correction Processing
In the all-region depth information correction processing of Step 6, a mean value of depth information on the parallax calculation regions F1 to F60 is first determined on a row-by-row basis. In a case where the depth information per parallax calculation region F1 to F60 is as shown in FIG. 23, for example, the first to the sixth region rows respectively have a mean value of 1.2, 3.6, 6.0, 7.2, 4.0 and 1.2.
Of the rows of parallax calculation regions, extracted is a row of regions wherein foreground object(s) occupy greater area than in the other region rows. That is, the region row having the greatest mean value of the depth information is extracted. According to the example of FIG. 23, the regions of the fourth row are extracted.
Subsequently, regions in rows below the extracted row are each subject to adjustment of the depth information thereof so that each of the regions is not sharply decreased in the depth information thereof relative to that of the region immediately thereabove. More specifically, of the regions included in the rows below the extracted row, a region, having depth information smaller than a region thereabove by not less than three, is changed in its depth information to a value smaller than that of the region thereabove by two.
In the example of FIG. 23, out of the regions F41 to F50 of the fifth row, the regions F42 to F49 having the depth information smaller than the respective regions thereabove by not less than three are first corrected in their depth information, as shown in FIG. 24. Subsequently, of the regions F51 to F60 in the sixth row, the regions F53 to F58 having the depth information smaller than those (after correction) of the respective regions thereabove by not less than three are corrected in the depth information thereof.
Where depth information has a relation with a vertical level of screen at an optional horizontal position as represented by a curve U1 in FIG. 25, the relation of the depth information versus the vertical level of screen is corrected to that represented by a curve U2 in FIG. 25.
It is for the following reason that, of the rows of parallax calculation regions, the region rows below the region row wherein foreground object(s) occupy greater area than in these rows are corrected in the depth information thereof.
In most cases, objects on the front of the scene are located on the lower side of the screen. In addition, images of objects located on the lower side of the screen, such as of ground and the like, often have a little variation. The image with a little variation like that of the ground has a low high-frequency component and therefore, has a low depth information value despite being located to the front of the scene. Hence, the depth correction is provided in order to increase the depth information value with respect to the image of the foreground object and with a low high-frequency component to a degree that the depth information value of such a region is not excessive of that of a region thereabove.
(7) Group-Boundary Depth Information Correction Processing
In some cases, regions in a boundary portion between two adjacent groups may not be divided into correct groups. Further if, in the boundary portion between the two adjacent groups, depth estimations of the respective groups have great difference from each other, a serious image distortion results.
Hence, the group-boundary depth information correction processing of Step 7 first checks a boundary portion between the respective pairs of adjacent groups to determine whether a difference between depth information of parallax calculation regions of one group and that of regions of the other group exceeds a predetermined value or not. If the difference in depth information between the two groups exceeds the predetermined value, the smaller depth information (or that of the parallax calculation regions corresponding to objects located relatively rearwardly) is increased so that the difference in depth information therebetween may be less than the predetermined value.
(8) In-Group Depth Information Correction Processing
The aforesaid correction processings of Steps 6 and 7 may result in production of difference in the depth information of regions of the same group. If this difference is great, a serious image distortion results. Therefore, the in-group depth information correction processing of Step 8 smoothes out depth estimations with respect to regions of each group.
It is assumed that a target region A has depth information HA while four regions U, D, L and R adjacent thereto and included in the same group with the region A have depth information Hu, HD, HL and HR, respectively, as shown in FIG. 26. Then, the depth estimation HA of the target region A is corrected based on the following equation (6):
HA=(HA/2)+{(HU+HD+HL+HR)/8}xe2x80x83xe2x80x83(6)
Depth information on the respective parallax calculation regions F1 to F60 thus obtained are normalized again within the range of between 1 and 10.
(9) Parallax Information Calculation Processing
In the parallax information calculation processing of Step 9, the depth information on the respective parallax calculation regions F1 to F60 is converted into parallax information on the respective regions F1 to F60.
More specifically, based on a predetermined relation between the depth information and the parallax information, the depth information on the regions F1 to F60 is converted into the parallax information on a per-region basis. The relation of the parallax information versus the depth information is inversely proportional, as shown by straight lines S1 or S2 in FIG. 27.
Referring to FIG. 27, the relation of the parallax information with the depth information represented by the straight line S1 is used for obtaining a 3D image with a relatively great stereoscopic effect. On the other hand, the relation of the parallax information with the depth information represented by the straight line S2 is used for obtaining a 3D image with a relatively small stereoscopic effect. The stereoscopic effect can be adjusted by adjusting the relation of the parallax information versus the depth information in a range of between the straight lines S1 and S2.
The parallax information per parallax calculation region thus obtained is supplied to the parallax control circuit 4 (see FIG. 1).
FIG. 28 diagrammatically illustrates essential configurations of the parallax control circuit and the optional pixel delay FIFO shown in FIG. 1. The description hereinafter will be made on the assumption that there are the parallax calculation regions E1 to E12, as shown in FIG. 2.
In FIG. 28, out of the optional pixel delays FIFOs 11 to 13 and 21 to 23, only the left-eye-image optional pixel delay FIFO 11 and the right-eye-image optional pixel delay FIFO 21 with respect to Y signal are shown. However, the other optional pixel delays FIFOs 12-13 and 22-23 have the same configuration and are subject to the same control and therefore, a description thereof is omitted.
It is to be noted that the parallax information calculated by the CPU 3 pertains a central position of the respective parallax calculation regions E1 to E12. The parallax control circuit 4 serves to obtain parallax information with respect to the respective pixel positions in the one-field screen based on the parallax information with respect to the central position of the respective parallax calculation regions E1 to E12. The readout addresses of the left-eye-image optional pixel delays FIFOs 11 to 13 and of the right-eye-image optional pixel delays FIFOs 21 to 23 are controlled based on the parallax information per pixel position in order to produce, from a 2D image signal per pixel position, a left eye image and a right eye image which have parallax according to the parallax information with respect to the pixel position.
The parallax information with respect to the respective pixel positions in the one-field screen is generated by means of a timing signal generating circuit 51, a parallax interpolation factor generating circuit 52, parallax information storage means 60, a parallax selection circuit 80, first to fourth multipliers 81 to 84 and an adder circuit 85.
The horizontal synchronizing signal Hsync and the vertical synchronizing signal Vsync of the input image signal are applied to the timing signal generating circuit 51. In addition, the clock signal CLK for detecting the horizontal address in each horizontal period is also applied to the timing signal generating circuit 51.
Based on the horizontal synchronizing signal Hsync, vertical synchronizing signal Vsync and clock signal CLK, the timing signal generating circuit 51 generates and outputs a horizontal address signal HAD indicative of an absolute horizontal position of the input image signal, a vertical address signal VAD indicative of an absolute vertical position of the input image signal, a relative horizontal position signal HPOS indicative of a relative horizontal position of the input image signal and a relative vertical position signal VPOS indicative of a relative vertical position of the input image signal.
Now, the relative horizontal position and the relative vertical position of the input image signal will hereinbelow be described.
As shown in FIG. 29, the parallax calculation regions E1 to E12 of FIG. 2 are defined in the following manner. A whole screen is divided into 20 regions (hereinafter referred to as xe2x80x9cfirst segment regionxe2x80x9d) arranged in 4 rows and 5 columns, as indicated by broken lines in FIG. 29. A rectangular area having four vertexes positioned at centers of upper left, upper right, lower left and lower right first segment regions, respectively, is divided into 12 regions (hereinafter referred to as xe2x80x9csecond segment regionxe2x80x9d) arranged in 3 rows and 4 columns. These 12 second segment regions are defined as the parallax calculation regions E1 to E12, respectively.
In the figure, a number of horizontal pixels of the first and second segment regions is denoted by xe2x80x9cmxe2x80x9d whereas a number of vertical pixels of the first and second segment regions is denoted by xe2x80x9cnxe2x80x9d. The relative horizontal position of the input image signal is denoted by any one of 0 to (mxe2x88x921) with xe2x80x9c0xe2x80x9d indicating a left end position and xe2x80x9c(mxe2x88x921)xe2x80x9d indicating a right end position of each first segment region. A relative vertical position of the input image signal is denoted by any one of 0 to (nxe2x88x921) with xe2x80x9c0xe2x80x9d indicating an upper end position and (nxe2x88x921) indicating a lower end position of each first segment region.
The relative horizontal position signal HPOS and the relative vertical position signal VPOS of the input image signal are supplied to the parallax interpolation factor generating circuit 52. Based on the relative horizontal position signal HPOS, relative vertical position signal VPOS and the following equations (7), the parallax interpolation factor generating circuit 52 generates and outputs a first parallax interpolation factor KUL, a second parallax interpolation factor KUR, a third parallax interpolation factor KDL and a fourth parallax interpolation factor KDR:
KUL={(mxe2x88x92HOPS)/m}xc3x97{(nxe2x88x92VPOS)/n}KUR=(HPOS/m)xc3x97{(Nxe2x88x92VPOS)/n}KDL={(mxe2x88x92HPOS)/m}xc3x97(VPOS/n) KDR=(HPOS/m)xc3x97(VPOS/n)xe2x80x83xe2x80x83(7)
Now referring to FIG. 30, description will be made on a basic concept of a method of generating the parallax information with respect to each pixel position in the one-field screen. It is assumed that a horizontal/vertical position (hereinafter referred to as xe2x80x9ctarget positionxe2x80x9d) indicated by the horizontal address signal HAD and the vertical address signal VAD is at Pxy in FIG. 30. How to obtain parallax information with respect to the target position Pxy will be described as below.
(1) First, out of the parallax information on the respective parallax calculation regions E1 to E12 calculated by the CPU 3, parallax information pieces on the parallax calculation regions E1 to E4 are first extracted as UL, UR, DL and DR, respectively. In this example, the regions E1 to E4 include four vertexes PE1 to PE4 of a first segment region with the target position Pxy at their respective centers. More specifically, the parallax information piece on the region E1, wherein the upper left vertex of the first segment region with the target position Pxy is located at the center thereof, is extracted as a first parallax information UL, the parallax information piece on the region E2 with the upper right vertex of the first segment region located at its center being extracted as a second parallax information UR, the parallax information piece on the region E5 with the lower left vertex of the first segment region located at its center as a third parallax information DL and the parallax information piece on the region E6 with the lower right vertex of the first segment region located at its center as a fourth parallax information DR.
It is to be noted that in a case where only one of the four vertexes of a first segment region with the target position is at the center of a parallax calculation region like when the first segment region is at the upper left corner of the screen, a parallax information piece on this parallax calculation region is extracted as the first to the fourth parallax information pieces UL, UR, DL, DR.
In a case where only lower two of the four vertexes of a first segment region with the target position are located at the respective centers of parallax calculation regions like when the first segment region adjoins an upper-left-corner first segment region on its right side, parallax information pieces on parallax calculation regions corresponding to the lower two vertexes of the first segment region are extracted also as parallax information pieces UL and UR on parallax calculation regions corresponding to the upper two vertexes thereof.
In a case where only right-hand two of the four vertexes of a first segment region with the target position are located at the respective centers of parallax calculation regions like when the first segment region is immediately under the upper-left-corner first parallax region, parallax information pieces on the parallax calculation regions corresponding to the two right-hand vertexes of the first segment region are extracted also as parallax information pieces UL, DL on parallax calculation regions corresponding to the two left-hand vertexes thereof.
In a case where only upper two of the four vertexes of a first segment region with the target position are located at the respective centers of parallax calculation regions like when the first segment region adjoins a lower-right-corner first segment region on its left side, parallax information pieces on the parallax calculation regions corresponding to the upper two vertexes of the first segment region are extracted also as parallax information pieces DL, DR on parallax calculation regions corresponding to the lower two vertexes thereof.
In a case where only left-hand two of the four vertexes of a first segment region with the target position are located at the respective centers of parallax calculation regions like when the first segment region is located immediately above the lower-right-corner first segment region, parallax information pieces on the parallax calculation regions corresponding to the two left-hand vertexes of the first segment region are extracted also as parallax information pieces UR, DR on parallax calculation regions corresponding to the two right-hand vertexes thereof.
(2) Next, there are determined the first to the fourth parallax interpolation factors KUL, KUR, KDL and KDR.
The first parallax interpolation factor KUL is given by a product of a ratio {(mxe2x88x92HPOS)/m} of a distance xcex94XR between the target position Pxy and a right side of a first segment region e including the target pixel Pxy to a horizontal width m of the first segment region e, and a ratio {(nxe2x88x92VPOS)/n} of a distance xcex94YD between the target pixel Pxy and a lower side of the first segment region e to a vertical width n of the first segment region e. That is, the first parallax interpolation factor KUL correspondingly increases as a distance between an upper left vertex PE1 of the first segment region e with the target position Pxy and the target position Pxy decreases.
The second parallax interpolation factor KUR is given by a product of a ratio (HPOS/m) of a distance xcex94XL between the target position Pxy and a left side of the first segment region e with the target position Pxy to the horizontal width m of the first segment region e, and a ratio {(nxe2x88x92VPOS)/n} of a distance xcex94YD between the target position Pxy and the lower side of the first segment region e to the vertical width n of the first segment region e. That is, the second parallax interpolation factor KUR correspondingly increases as a distance between an upper right vertex PE2 of the first segment region e and the target pixel position Pxy decreases.
The third parallax interpolation factor KDL is given by a product of a ratio {(mxe2x88x92HPOS)/m} of a distance xcex94XR between the target position Pxy and the right side of the first segment region e with the target position Pxy to the horizontal width m of the first segment region e, and a ratio (VPOS/n) of a distance xcex94YU between the target position Pxy and an upper side of the first segment region e to the vertical width n of the first segment region e. That is, the third parallax interpolation factor KDL correspondingly increases as a distance between a lower left vertex PE5 of the first segment region e and the target position Pxy decreases.
The fourth parallax interpolation factor KDR is given by a product of a ratio (HPOS/m) of a distance xcex94XL between the target position Pxy and the left side of the first segment region e with the target position Pxy to the horizontal width m of the first segment region e, and a ratio (VPOS/n) of a distance xcex94YU between the target position Pxy and the upper side of the first segment region e to the vertical width n of the first segment region e. That is, the fourth parallax interpolation factor KDR correspondingly increases as a distance between a lower right vertex PE6 of the first segment region e and the target position Pxy decreases.
(3) The first to the fourth parallax information pieces UL, UR, DL, DR extracted in the aforesaid step (1) are respectively multiplied by the first to the fourth parallax interpolation factors KUL, KUR, KDL, KDR given in the aforesaid step (2). Then, parallax information on the target position Pxy is generated by summing up the resultant four products.
The parallax information storage means 60 includes first to twelfth parallax registers 61 to 72 in corresponding relation with the regions E1 to E2, respectively. The first to twelfth parallax registers 61 to 72 respectively store the parallax information per region E1 to E12 generated by the CPU 3.
The parallax selection circuit 80 is provided rearwardly of the parallax information storage means 60. The parallax selection circuit 80 is supplied with the parallax information by the parallax registers 61 to 72, respectively. Furthermore, the parallax selection circuit 80 is supplied with the horizontal address signal HAD and the vertical address signal VAD by the timing signal generating circuit 51.
According to a rule shown in FIG. 31(a), the parallax selection circuit 80 selectively outputs parallax information on a region corresponding to a horizontal address signal HAD and a vertical address signal VAD (in the example shown in FIG. 30, the parallax calculation region having the upper left vertex of the first segment region with the target position located at the center thereof) as the first parallax information UL. Further according to a rule shown in FIG. 31(b), the parallax selection circuit 80 selectively outputs parallax information on a region corresponding to a horizontal address signal HAD and a vertical address signal VAD in the example of FIG. 30, the parallax calculation region having the upper right vertex of the first segment region with the target position located at the center thereof) as the second parallax information UR.
According to a rule shown in FIG. 31(c), the parallax selection circuit 80 selectively outputs parallax information on a region corresponding to a horizontal address signal HAD and a vertical address signal VAD (in the example of FIG. 30, the parallax calculation region having the lower left vertex of the first segment region with the target position located at the center thereof) as the third parallax information DL. Further according to a rule shown in FIG. 31(d), the parallax selection circuit 80 selectively outputs parallax information on a region corresponding to a horizontal address signal HAD and a vertical address signal VAD (in the example of FIG. 30, the parallax calculation region having the lower right vertex of the first segment with the target position located at the center thereof) as the fourth parallax information DR. In FIG. 31, a representation xe2x80x9caxcx9cbxe2x80x9d like xe2x80x9c0xcx9cmxe2x80x9d denotes a value of not less than xe2x80x9caxe2x80x9d and less than xe2x80x9cbxe2x80x9d.
The first parallax information UL, the second parallax information UR, the third parallax information DL and the fourth parallax information DR are applied to the first to the fourth multipliers 81 to 84, respectively.
The first to the fourth multipliers 81 to 84 also receives from the parallax interpolation factor generating circuit 52 the first parallax interpolation factor KUL, the second parallax interpolation factor KUR, the third parallax interpolation factor KDL and the fourth parallax interpolation factor KDR, respectively.
The first multiplier 81 multiplies the first parallax information UL by the first parallax interpolation factor KUL. The second multiplier 82 multiplies the second parallax information UR by the second parallax interpolation factor KUR. The third multiplier 83 multiplies the third parallax information DL by the third parallax interpolation factor KDL. The fourth multiplier 84 multiplies the fourth parallax information DR by the fourth parallax interpolation factor KDR.
Outputs of these multipliers 81 to 84 are added together by the adder circuit 85 for generating parallax information PR on the target position.
The optional pixel delays FIFOs 11 and 21 each include a pair of line memories 11a-11b or 21a-21b for performing horizontal phase control on a basis smaller than one pixel. The respective pairs of line memories 11a-11b and 21a-21b of the optional pixel delays FIFOs 11, 21 are supplied with Y signal and the clock signal CLK.
The horizontal address signal HAD outputted from the timing signal generating circuit 51 is also applied to a standard address generating circuit 90. The standard address generating circuit 90 generates and outputs a standard write address WAD and a standard readout address RAD with respect to the line memories 11a-11b and 21a-21b of the optional pixel delays FIFOs 11 and 21. Further, the standard address generating circuit 90 also outputs a synchronizing signal Csync added to the left-eye image signal and the right-eye image signal which are generated by the 2D/3D image conversion system. A horizontal synchronizing signal represented by the synchronizing signal Csync is delayed relative to the horizontal synchronizing signal Hsync of the input image signal by a predetermined number of clocks.
The standard readout address RAD is delayed relative to the standard write address WAD by a predetermined number of clocks so that the horizontal phase of the image signal inputted in the respective optional pixel delays FIFOs 11, 21 may be advanced or lagged relative to a reference horizontal phase defined by the standard readout address. The standard write address WAD outputted from the standard address generating circuit 90 is applied to the respective pairs of line memories 11a-11b and 21a-21b of the optional pixel delays FIFOs 11, 21 as a write control signal indicative of an address to write to.
The standard readout address RAD from the standard address generating circuit 90 is applied to an adder 91 and a subtractor 92. The adder 91 and the subtractor 92 are also supplied with the parallax information PR on the target position which is supplied from the adder circuit 85.
The adder 91 serves to add the parallax information PR to the standard readout address RAD, thereby giving a left-eye image readout address PRL.
An integer portion PRL1 of the left-eye image readout address PRL is applied, as a readout address RADL1, to the first line memory 11a of the left-eye image optional pixel delay FIFO 11. This permits Y signal to be read out from an address of the first line memory 11a in correspondence to the address RADL1. The Y signal thus read out is applied to a first left-eye image multiplier 101.
An address value given by adding 1 to the integer portion PRL1 of the left-eye image readout address PRL is applied as a readout address RADL2 to the second line memory 11b of the left-eye image optional pixel delay FIFO 11. This permits Y signal to be read out from an address of the second line memory 11b in correspondence to the address RADL2. The Y signal thus read out is applied to a second left-eye image multiplier 102.
The readout address RADL1 with respect to the first line memory 11a differs from the readout address RADL2 with respect to the second line memory 11b by xe2x80x9c1xe2x80x9d and therefore, a Y signal read out from the first line memory 11a has a horizontal position shifted by xe2x80x9c1xe2x80x9d from that of a Y signal read out from the second line memory 11b. 
A decimal fraction portion PRL2 of the left-eye image readout address PRL is applied, as a second left-eye image interpolation factor, to the second left-eye image multiplier 102. A value (1xe2x88x92PRL2) is obtained by subtracting the decimal fraction portion PRL2 of the left-eye image readout address PRL from 1 so as to be applied, as a first left-eye image interpolation factor, to the first left-eye image multiplier 101.
Thus, the first left-eye image multiplier 101 multiplies the Y signal read out from the first line memory 11a by the first left-eye image interpolation factor (1xe2x88x92PRL2). The second left-eye image multiplier 102 multiplies the Y signal read out from the second line memory 11b by the second left-eye image interpolation factor PRL2. The resultant Y signals thus given by these multipliers 101, 102 are added together by an adder 103 and then outputted as a left-eye image Y signal YL-OUT.
Thus is obtained the left-eye image Y signal, the horizontal phase of which is lagged relative to the reference horizontal phase defined by the standard readout address RAD by a value based on the parallax information on the target position.
The subtractor 92 serves to subtract the parallax information PR from the standard readout address RAD, thereby giving a right-eye image readout address PRR.
An integer portion PRR1 of the right-eye image readout address PRR is applied, as a readout address RADR1, to the first line memory 21a of the right-eye image optional pixel delay FIFO 21. This permits Y signal to be read out from an address of the first line memory 21a in correspondence to the address RADR1. The Y signal thus read out is applied to a first right-eye image multiplier 111.
An address value is given by adding 1 to the integer portion PRR1 of the right-eye image readout address PRR and applied as a readout address RADR2 to the second line memory 21b of the right-eye image optional pixel delay FIFO 21. This permits Y signal to be read out from an address of the second line memory 21b in correspondence to the address RADR2. The Y signal thus read out is applied to a second right-eye image multiplier 112.
The readout address RADR1 with respect to the first line memory 21a differs from the readout address RADR2 with respect to the second line memory 21b by xe2x80x9c1xe2x80x9d and therefore, the Y signal read out from the first line memory 21a has a horizontal position shifted by xe2x80x9c1xe2x80x9d from that of the Y signal read out from the second line memory 21b. 
A decimal fraction portion PRR2 of the right-eye image readout address PRR is applied, as a second right-eye image interpolation factor, to the second right-eye image multiplier 112. A value (1xe2x88x92PRR2) is obtained by subtracting the decimal fraction portion PRR2 of the right-eye image readout address PRR from 1 so as to be applied, as a first right-eye image interpolation factor, to the first right-eye image multiplier 111.
Thus, the first right-eye image multiplier 111 serves to multiply the Y signal read out from the first line memory 21a by the first right-eye image interpolation factor (1xe2x88x92PRR2). The second right-eye image multiplier 112 serves to multiply the Y signal read out from the second line memory 21b by the second right-eye image interpolation factor PRR2. The resultant Y signals thus given by these multipliers 111, 112 are added together by an adder 113 and then outputted as a right-eye image Y signal YR-OUT.
Thus is obtained the right-eye image Y signal, the horizontal phase of which leads relative to the reference horizontal phase defined by the standard readout address RAD by a value based on the parallax information on the target position.
FIG. 32 is a timing chart showing signals at the respective addresses in a case where the parallax information on the target position is 0. Where the parallax information is 0, the left-eye image readout address PRL outputted from the adder 91 and the right-eye image readout address PRR outputted from the subtractor 92 are both equal to the standard readout address RAD, consisting only of the integer portion without the decimal fraction portion.
Therefore, the readout address RADL1 with respect to the first line memory 11a of the left-eye image optional pixel delay FIFO 11 and the readout address RADR1 with respect to the first line memory 21a of the right-eye image optional pixel delay FIFO 21 are both equal to the standard readout address RAD.
On the other hand, the readout address RADL2 with respect to the second line memory 11b of the left-eye image optional pixel delay FIFO 11 and the readout address RADR2 with respect to the second line memory 21b of the right-eye image optional pixel delay FIFO 21 are both at a value greater than the standard readout address RAD by one.
The first left-eye image interpolation factor (1xe2x88x92PRL2) and the first right-eye image interpolation factor (1xe2x88x92PRR2) are both at a value of 1 whereas the second left-eye image interpolation factor PRL2 and the second right-eye image interpolation factor PRR2 are both at a value of 0.
As a result, Y signal read out from the address of the first line memory 11a of the left-eye image optional pixel delay FIFO 11 in correspondence with the standard address RAD is outputted from the adder 103 as the left-eye image Y signal YL-OUT, whereas Y signal read out from the address of the first line memory 21a of the right-eye image optional pixel delay FIFO 21 in correspondence with the standard address RAD is outputted from the adder 113 as the right-eye image Y signal YR-OUT. That is, the two Y signals having the same horizontal phase shift value or the two Y signals without parallax are outputted as the left-eye image Y signal and the right-eye image Y signal, respectively.
FIG. 33 is a block diagram showing exemplary address values in a case where a standard write address with respect to a certain target position is 20, the standard readout address with respect to the target position is at 10 and the parallax information on the target position is at 1.2. FIG. 34 is a timing chart showing signals at the respective addresses in this example.
In this case, a left-eye image readout address PLR outputted from the adder 91 is 11.2, the integer portion PRL1 of which is 11 and the decimal fraction portion PRL2 of which is 0.2.
Accordingly, a readout address RADL1 with respect to the first line memory 11a of the left-eye image optional pixel delay FIFO 11 is 11 whereas a readout address RADL2 with respect to the second line memory 11b is 12. On the other hand, a first left-eye image interpolation factor KL1{=(1xe2x88x92PRL2)} is 0.8 whereas a second left-eye image interpolation factor KL2(=PRL2) is 0.2.
Thus, a Y signal (Y11) is read out from the address 11 of the first line memory 11a of the left-eye image optional pixel delay FIFO 11 so that the first multiplier 101 outputs a signal (0.8*Y11) given by multiplying the read Y signal (Y11) by 0.8.
On the other hand, a Y signal (Y12) is read out from the address 12 of the second line memory 11b of the left-eye image optional pixel delay FIFO 11 so that the second multiplier 102 outputs a signal (0.2*Y12) given by multiplying the read Y signal (Y12) by 0.2. Then, the adder 103 outputs a left-eye image Y signal YL-OUT equal to 0.8*Y11+0.2*Y12. In other words, the Y signal equal to the readout address of 11.2 is outputted as the left-eye image Y signal YL-OUT.
The subtractor 92 outputs a right-eye image readout address PRR of 8.8, the integer portion of which is 8 and the decimal fraction portion of which is 0.8.
Accordingly, the readout address RADR1 with respect to the first line memory 21a of the right-eye image optional pixel delay FIFO 21 is at 8 whereas the readout address RADR2 with respect to the second line memory 21b is at 9. On the other hand, the first right-eye image interpolation factor KR1{=(1xe2x88x92PPR2)} is 0.2 whereas the second right-eye image interpolation factor KR2(=PRR2) is 0.8.
Thus, a Y signal(Y8) is read out from the address 8 of the first line memory 21a of the right-eye image optional pixel delay FIFO 21 so that the first multiplier 111 outputs a signal (0.2*Y8) given by multiplying the read Y signal (Y8) by 0.2.
On the other hand, a Y signal (Y9) is read out from the address 9 of the second line memory 21b of the right-eye image optional pixel delay FIFO 21 so that the second multiplier 112 outputs a signal (0.8*Y9) given by multiplying the read Y signal (Y9) by 0.8. Then, the adder 113 outputs a right-eye image Y signal YR-OUT equal to 0.2*Y8+0.8*Y9. In other words, the Y signal equal to the readout address of 8.8 is outputted as the right-eye image Y signal YR-OUT.
As a result, there are obtained left-eye and right-eye images each having a parallax of 11.2xe2x88x928.8=2.4, or twice the parallax information piece of 1.2.
In the aforementioned 2D/3D still image conversion method, there may sometimes be a great parallax value between regions in the one-field screen or the parallax value may sometimes vary greatly between fields. This results in poor visibility of a resultant 3D image.
It is therefore, an object of the invention to provide a 2D/3D image conversion system offering 3D images suffering less image distortion.
By the way, a 2D image signal to be converted into a 3D image signal includes both a moving image portion and a still image portion. Hence, it may be designed that the moving image portion is subjected to the 2D/3D moving image conversion process for generating a 3D image while the still image portion is subjected to the 2D/3D still image conversion process for generating a 3D image.
More specifically, a motion vector is sensed from the 2D image to be converted. If a value of a horizontal motion vector exceeds a predetermined value, a currently inputted image is determined to be a moving image and subjected to the 2D/3D moving image conversion process for conversion into the 3D image. If, on the other hand, the value of the horizontal motion vector is below the predetermined value, the currently inputted image is determined to be a still image and subjected to the 2D/3D still image conversion process for conversion into the 3D image.
According to such a conversion process, however, the parallax varies greatly when the input image is switched between the moving image and the still image. This results in an image which detrimentally appears incongruous to an observer.
In view of the foregoing, it is an object of the invention to provide a 2D/3D image conversion system and method which utilize the 2D/3D moving image conversion method and the 2D/3D still image conversion method in combination for converting two-dimensional images into three-dimensional images in a manner such that the parallax variations are decreased when the input images are switched between the moving image and the still image.
The aforesaid 2D/3D still image conversion method utilizes the image characteristic value as well as the background weight for generation of the depth information per group. The background weight is a numerical value preset per parallax calculation region in order to enhance precisions of the depth information. If, for example, a more front view has a correspondingly greater depth information value while a more rear view has a correspondingly smaller depth information value, the background weights for the respective parallax calculation regions F1 to F60 (see FIG. 13) generally have values such that a central portion of the image field is greater than a peripheral portion thereof while a lower portion thereof is greater than an upper portion thereof, as shown in FIG. 49. This is because objects in the front view range are placed at the central portion rather than the peripheral portion or at the lower portion rather than the upper portion of the typical image frame.
However, in an image of an exit of a tunnel or an image viewed through an endscope, the front view occupies the peripheral portion of the image and the rear view occupies the central portion. Hence, in this case, incorrect depth information is generated, resulting in an incorrect 3D image.
Alternatively, there is known a method wherein the motion vector per parallax calculation region is generated from the 2D image signal and used for generating the depth information. Unfortunately however, the method of generating depth information by the use of the motion vector is difficult to apply to still images with no motion.
It is therefore, an object of the invention to provide a 2D/3D image conversion system and method of converting two-dimensional images into three-dimensional images, which system and method are adapted to generate depth information increased in precision.
A first 2D/3D image conversion system according to the invention comprises: depth information generating means which extracts from a 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, and generates depth information per parallax calculation region based on the extracted image characteristic value; target phase value calculating means for calculating a target phase value per parallax calculation region based on a dynamic range of a preset target phase value and on depth information generated per given unit area by the depth information generating means;
real phase value calculating means which calculates a current real phase value of each parallax calculation region in a manner to progressively approximate a current real phase value to a target phase value of a parallax calculation region with a corresponding previous real phase value, and then determines a current real phase value per given unit area in the one-field screen based on the current real phase value per parallax calculation region thus obtained; and phase control means for generating a first image signal and a second image signal from a signal residing in each given unit area of the 2D input image, the first and second image signals having a horizontal phase difference therebetween based on the current real phase value corresponding to the given unit area.
A second 2D/3D image conversion system according to the invention comprises: depth information generating means which extracts from a 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, and generates depth information per parallax calculation region based on the extracted image characteristic value; correction means for correcting a dynamic range of a preset target phase value thereby decreasing the dynamic range of the target phase value when a maximum value of depth information differences between respective pairs of adjacent regions exceeds a reference value; target phase value calculating means for calculating a target phase value per parallax calculation region based on the dynamic range of the target phase value given by the correction means and on depth information per given unit area generated by the depth information generating means; real phase value calculating means which calculates a current real phase value per parallax calculation region in a manner to progressively approximate a current real phase value to a target phase value of a parallax calculation region with a corresponding previous real phase value, and determines a current real phase value per given unit area in the one-field screen based on the current real phase value per parallax calculation region thus obtained; and phase control means for generating a first image signal and a second image signal from a signal residing in each given unit area of the 2D input image, the first and second image signals having a horizontal phase difference therebetween based on the current real phase value corresponding to the given unit area.
A third 2D/3D image conversion system according to the invention comprises: image converting means which produces from a 2D input image a main image serving as a base and a sub-image time-delayed relative to the main image, and outputs either one of the main image and the sub-image as a left eye image and the other as a right eye image; moving image/still image judging means for determining whether the 2D input image is a moving image or a still image; still image rate calculating means for calculating a still image rate based on a judgment result given by the moving image/still image judging means; depth information generating means which extracts from the 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, and generates depth information per parallax calculation region based on the extracted image characteristic value; correction means for correcting a dynamic range of a preset target phase value by multiplying the dynamic range of the target phase value by the still image rate; target phase value calculating means for calculating a target phase value per parallax calculation region based on the dynamic range of the target phase value corrected by the correction means and on depth information per given unit area generated by the depth information generating means; real phase value calculating means which calculates a current real phase value per parallax calculation region in a manner to progressively approximate a current real phase value to a target phase value of a parallax calculation region with a corresponding previous real phase value, and determines a current real phase value per given unit area in the one-field screen based on the current real phase value per parallax calculation region thus obtained; and phase control means for performing a horizontal phase control on a signal residing in each given unit area of the right eye image and the left eye image which are outputted by the image converting means, the horizontal phase control being based on a real phase value of the given unit area supplied by the real phase value calculating means.
A fourth 2D/3D image conversion system according to the invention comprises: image converting means which produces from a 2D input image a main image serving as a base and a sub-image time-delayed relative to the main image, and outputs either one of the main image and the sub-image as a left eye image and the other as a right eye image; moving image/still image judging means for determining whether the 2D input image is a moving image or a still image; still image rate calculating means for calculating a still image rate based on a judgment result given by the moving image/still image judging means; depth information generating means which extracts from the 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, and generates depth information per parallax calculation region based on the extracted image characteristic value; first correction means for correcting a dynamic range of a preset target phase value thereby decreasing the dynamic range of the target phase value when a maximum value of depth information differences between respective pairs of adjacent regions exceeds a reference value; second correction means for correcting the dynamic range of the target phase value by multiplying the dynamic range of the target phase value, given by the first correction means, by the still image rate; target phase value calculating means for calculating a target phase value per parallax calculation region based on the dynamic range of the target phase value given by the second correction means and on depth information per given unit area generated by the depth information generating means; real phase value calculating means which calculates a current real phase value per parallax calculation region in a manner to progressively approximate a current real phase value to a target phase value of a parallax calculation region with a corresponding previous real phase value, and determines a current real phase value per given unit area in the one-field screen based on the current real phase value per parallax calculation region thus obtained; and phase control means for performing a horizontal phase control on a signal residing in each given unit area of the right eye image and the left eye image which are outputted by the image converting means, the horizontal phase control being based on a real phase value of the given unit area supplied by the real phase value calculating means.
The aforesaid still image rate calculating means may include, for example, means for setting the still image rate to xe2x80x9c0xe2x80x9d when the 2D input image is determined to be a moving image by the moving image/still image judging means while setting the still image rate to xe2x80x9c100xe2x80x9d when the 2D input image is determined to be a still image, and means for smoothing the set still image rates between fields.
The aforesaid real phase value calculating means may, for example, calculate an amount of image motion per parallax calculation region based on the 2D input image so as to calculate a current real phase value per parallax calculation region in a manner such that a current real phase value of a parallax calculation region with greater image motion is correspondingly more quickly approximated to a target phase value of the region, and then determine a current real phase value per given unit area in the one-field screen based on the current real phase value per parallax calculation region thus obtained.
A fifth 2D/3D image conversion system according to the invention comprises: image converting means which produces from a 2D input image a main image serving as a base and a sub-image time-delayed relative to the main image, and outputs either one of the main image and the sub-image as a left eye image and the other as a right eye image; moving image/still image judging means for determining whether the 2D input image is a moving image or a still image; still image rate calculating means for calculating a still image rate based on a judgment result given by the moving image/still image judging means; parallax information generating means which extracts from the 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, and generates parallax information per parallax calculation region based on the extracted image characteristic value; parallax information correcting means which corrects the parallax information per parallax calculation region by multiplying the parallax information per parallax calculation region, generated by the parallax information generating means, by the still image rate given by the still image rate calculating means, and generates parallax information per given unit area in the one-field screen based on each parallax information piece thus corrected; and phase control means for performing a horizontal phase control on a signal residing in each given unit area of the right eye image and the left eye image which are outputted by the image converting means, the horizontal phase control being based on the parallax information of the given unit area supplied by the parallax information correcting means.
The still image rate calculating means may include, for example, means for setting the still image rate to xe2x80x9c0xe2x80x9d when the 2D input image is determined to be a moving image while setting the still image rate to xe2x80x9c100xe2x80x9d when the 2D input image is determined to be a still image, and means for smoothing the set still image rates between fields.
A first method of converting a 2D image into a 3D image according to the invention comprises the steps of: a first step of producing from a 2D input image a main image serving as a base and a sub-image time-delayed relative to the main image, followed by outputting either one of the main image and the sub-image as a left eye image and the other as a right eye image; a second step of determining whether the 2D input image is a moving image or a still image; a third step of calculating a still image rate based on the determination given by the second step; a fourth step of extracting from the 2D input image a perspective image characteristic value of each of a plurality of parallax calculation regions defined in a one-field screen, followed by generating parallax information per parallax calculation region based on the extracted image characteristic value; a fifth step of correcting the parallax information per parallax calculation region by multiplying the parallax information per parallax calculation region generated at the fourth step by the still image rate given by the third step, followed by generating parallax information per given unit area in the one-field screen based on each parallax information piece thus corrected; and a sixth step of performing a horizontal phase control on a signal residing in each given unit area of the right eye image and the left eye image which are outputted at the first step, the horizontal phase control being based on the parallax information of the given unit area supplied by the fifth step.
The third step may include, for example, a step of setting the still image rate to xe2x80x9c0xe2x80x9d when the 2D input image is determined by the second step to be a moving image while setting the still image rate to xe2x80x9c100xe2x80x9d when the 2D input image is determined to be a still image, and a step of smoothing the still image rates between fields.
A sixth system for converting a 2D image into a 3D image comprises: depth information generating means for generating depth information from a 2D input image signal on a field-by-field basis, the depth information indicating a perspective image characteristic of each of a plurality of parallax calculation regions defined in a one-field screen; parallax information generating means for generating parallax information per parallax calculation region based on the depth information; and parallax control means which controls the 2D input image signal based on the parallax information in a manner to produce parallax between a left eye image and a right eye image, and produces a left-eye image signal and a right-eye image signal; the depth information generating means including: (a) image characteristic value generating means for generating from the 2D input image signal an image characteristic value of each parallax calculation region; (b) group information generating means which divides, based on the image characteristic value, all the parallax calculation regions in the one-field screen into groups associated with respective objects included in the field, and generates group information indicating to which group each parallax calculation region belongs; (c) background weight generating means for generating a background weight per parallax calculation region; and (d) depth information calculating means which combines the image characteristic value and the background weight on a group-by-group basis by using said image characteristic value, group information and background weight, thereby calculating the depth information; the background weight generating means including optionally selected one of the following means or an optional combination thereof: (a) background-weight data receiving means for receiving from an external source background weight data including the background weight per parallax calculation region; (b) background-weight data selection means for selectively reading out a desired background weight from background-weight data storage means previously storing one or plural pieces of background weight data; and (c) view range judging means which generates a motion vector from the 2D input image signal for using the generated motion vector for determination of a view range of each parallax calculation region, and generates a background weight based on the determination of the view range.
A second method of converting a 2D image into a 3D image comprises the steps of generating depth information from a 2D input image signal on a field-by-field basis, the depth information indicating a perspective image characteristic of each of a plurality of parallax calculation regions defined in a one-field screen; generating parallax information per parallax calculation region based on the depth information thus generated; and generating a left-eye image signal and a right-eye image signal based on the parallax information, the method wherein the generation of the depth information is accomplished through the steps of generating from the 2D input image signal an image characteristic value of each parallax calculation region; dividing, based on the image characteristic value, all the regions in the one-field screen into groups associated with respective objects included in the field for generation of group information indicating to which group each parallax calculation region belongs; generating a background weight per parallax calculation region; and combining the image characteristic value and the background weight on a group-by-group basis by using the image characteristic value, group information and background weight, thereby calculating the depth information, the generation of the background weight being performed by optionally selected one of the following methods or an optional combination thereof which include a method of receiving from an external source background weight data including background weights of the respective parallax calculation regions; a method of selectively reading out a desired background weight from a plurality of background-weight data pieces previously stored; and a method of determining a view range of each parallax calculation region based on a motion vector and generating a background weight based on the determination of the view range.